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Abstract 

Random hypothesis samphng hes at the core of many popular robust fitting 
techniques such as RANSAC. In this paper, we propose a novel hypothe- 
sis sampling scheme based on incremental computation of distances between 
partial rankings (top-fc lists) derived from residual sorting information. Our 
method simultaneously (1) guides the sampling such that hypotheses corre- 
sponding to all true structures can be quickly retrieved and (2) filters the 
hypotheses such that only a small but very promising subset remain. This 
permits the usage of simple agglomerative clustering on the surviving hy- 
potheses for accurate model selection. The outcome is a highly efficient 
multi-structure robust estimation technique. Experiments on synthetic and 
real data show the superior performance of our approach over previous meth- 
ods. 
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1. Introduction 

Robust model fitting techniques play an integral role in computer vision 
since the observations or measurements are frequently contaminated with 
outliers. Major applications include the estimation of various projective en- 
tities from multi-view data \V\ which often contain false correspondences. At 
the core of many robust techniques is random hypothesis generation, i.e., 
iteratively generate many hypotheses of the geometric model from randomly 
sampled minimal subsets of the data. The hypotheses are then scored ac- 
cording to a robust criterion (e.g., RANSAC [2j) or clustered (e.g.. Mean 
Shift [3j) to find the most promising model(s). Success rests upon retrieving 
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(a) (b) (c) 

Figure 1: (a) Input data with 5 structures (lines) with 100 points per structure and 250 
gross outhers. The inher scale is 0.01. (b) 500 hypotheses are generated with the proposed 
multi- structure guided sampling scheme and simultaneous hypothesis filtering, producing 
180 good hypotheses as shown in the figure, (c) Simple agglomerative clustering of the 
remaining 180 hypotheses gives the final fitting result. 

an adequate number of all-inlier minimal subsets which may require a large 
enough number of sampling steps. 

This paper addresses two major issues affecting the current paradigm of 
robust estimation. The first is that hypothesis generation tends to be time 
consuming for heavily contaminated data. Previous methods attempted to 
improve sampling efficiency by guiding the sampling such that the probabil- 
ity of selecting all-inlier minimal subsets is increased. These methods often 
depend on assumptions or domain knowledge of the data, e.g., inliers have 
higher keypoint matching scores [JJ [5] or are correspondences that respect 
local geometry patterns [6j. Most methods, however, are not optimized for 
data with multiple instances (or structures f^) of the geometric model. This 
is because they sample based on estimated inlier probabilities alone while 
ignoring the fact that only inliers from the same structure should be in- 
cluded in the same minimal subset. Such methods may inefficiently generate 
a large number of samples before obtaining an all-inlier minimal subset for 
each genuine structure in the data. 

The second crucial issue is the lack of a principled approach to fit the 
multiple structures in the data. Many previous works |Sl E] simply apply 
RANSAC sequentially, i.e., fit one structure, remove corresponding inliers, 
then repeat. This is risky because inaccuracies in the initial fits will be 
amplified in the subsequent fits [lOj. Moreover, finding a stopping criterion 
for sequential fitting that accurately reflects the true number of structures 
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is non-trivial. Methods based on clustering [TT] or mode detection [3l [12] 
given the generated hypotheses are not affected by the dangers of sequential 
fitting. However, if there are insufficient hypotheses corresponding to the true 
structures, the genuine clusters will easily be overwhelmed by the irrelevant 
hypotheses. Consequently, these methods often miss the true structures or 
find spurious structures. 

The inability to retrieve "good" hypotheses at sufficiently large quanti- 
ties represents the fundamental obstacle to the satisfactory performance of 
previous methods. To address this limitation, we propose a novel hypothe- 
sis sampling scheme based on incremental computation of distances between 
partial rankings or top-k lists [13] derived from residual sorting information. 
Our approach enhances hypothesis generation in two ways: (1) The com- 
puted distances guide the sampling such that inliers from a single coherent 
structure are more likely to be simultaneously selected. This dramatically 
improves the chances of hitting all-inlier minimal subsets for each structure in 
the data. (2) The qualities of the generated hypotheses are evaluated based 
on the computed distances. This permits an on-the-ffy filtering scheme to 
reject "bad" hypotheses. The outcome is a set of only the most promising 
hypotheses which facilitate a simple agglomerative clustering step to fit all 
the genuine structures in the data. Fig. [T] summarizes the proposed approach. 

The rest of the paper is organized as follows: Sec. [2] describes how to de- 
rive data similarities from residual sorting information by comparing top-k 
lists. Sec. |3] describes our guided sampling scheme with simultaneous hy- 
pothesis filtering and incremental computations of distances between top-k 
lists. Sec. |4] describes how multi-structure fitting can be done by a sim- 
ple agglomerative clustering on the promising hypotheses returned by our 
method. Sec. [5] presents results on synthetic and real data which validate 
our approach. Finally, we draw conclusions in Sec. |6| 

2. Data Similarity by Comparing Top-fc Lists 

A key ingredient of our guided sampling scheme is a data similarity mea- 
sure. This section describes how to derive such a measure from residual 
sorting information. 

2.1. Top-k Lists from Residual Sorting Information 

We measure the similarity between two input data based on the idea 
that if they are inliers from the same structure, then their preferences to the 
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hypotheses as measured by residuals wiU be similar. Such preferences can be 
effectively captured by lists of ranked residuals. 

Let X = {xi}^^ be a set of input data and B = {9j}j^i a set of M 
hypotheses, where each hypothesis Oj is fitted from a minimal subset of p 
data (e.g., p=2 for line fitting). For each datum x^, we compute its absolute 
residual as measured to M hypotheses to form a residual vector 

n = [r?,r«,...,rg]. (1) 

We sort the elements in to obtain a sorted residual vector 



r^ = ^%r--,r%\ (2) 



such that r^(.) < - - - < ly The permutation [A^ , • • • , A^] encapsulate the 

data preference of Xi to the hypotheses, i.e., Xi is more likely to be an inlier 
to the hypotheses which have higher rank. 

The top-k list of data Xi is defined as the first k elements in the permu- 
tation [A^'^^ • • • , A^] , i.e., 

n = [Af\---,A«]. (3) 

The top-k list essentially gives the top-k hypotheses preferred by Xi. 

Fig [2] illustrate our idea to measure the data similarity using data pref- 
erences to the hypotheses. For data Xi and X2, they are inliers from the 
same structure and their corresponding top- 10 lists ri and r2 are similar, 
e.g., hypotheses 6,7 and 8 are highly ranked. For two inliers Xi and Xs from 
different structure, their corresponding top- 10 lists ri and ts are inconsistent, 
e.g., hypotheses 2 and 4 are highly ranked by Xs but not by Xi. 

2.2. The Spearman Footrule Distance 

Given the top-fc lists, we measure their similarity using the Spearman 
Footrule (SF) distance [13]. Let r be a top-k list and be a set of elements 
contained in r. Denote the position of the element m ^ m r hy T{m). 
The SF distance between two iop-k lists and r^/ is defined as 

FW(r„r,) = KM-^'Mh (4) 

where ^ > is the so-called location parameter (often set to + 1), r/(m) = 
Ti{m) if m G D^.] otherwise r/(m) = ^, and t[, is similarly obtained from r^/. 
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Figure 2: Top Left: 10 hypotheses are randomly generated using input data in Fig. 1(a) 



Top Right: 3 data are selected {xi and X2 are inliers from the same structure; xs is an 
inlier from different structure). Bottom: top-10 hypotheses corresponds to the selected 
data. (Best viewed in color) 



2.3. Measuring Similarity between Data 

To measure the similarity between two data, we use the SF distance 
(Eq. [4]) between their corresponding top-k hsts. The similarity value between 
two data x,- and x,/ is defined as 



(5) 



Note that we normahze F^^\Ti^ r^/) such that rf(r^, r^/) is between (dissimilar) 
and 1 (identical) . By comparing the top-/c lists between all data, we obtain a 
N X N similarity matrix K with 



where K{i^ i') denotes the element at its i-th row and i'-ih column. 
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Fig. |3(a)| shows an example of which is generated from the input 

The evident block structures in K correspond 
As shown in Fig. |3(b)| across various fc, the 



data shown in Fig. 1(a) 
to the 5 lines in Fig. 



1(a) 



average of similarity values between two inliers from the same structure (SS) 
is higher than that from different structure (DS). It supports the observation 



(a) 



(b) 



Figure 3: (a) Similarity matrix K for data shown in Fig. |l(a)| (data is arranged according 
to structure membership for representation only) (b) Under various k, the average of 
similarity values between two inliers from the same structure (SS), two inliers from different 
structures (DS), an inlier and a gross outlier (10), two gross outliers (00). 



from our given example (Fig. [2]) that inliers from the same structure gives 
consistent preferences to the hypotheses. From Fig. 3(b)| it also shows that 
the inconsistent preferences to hypotheses occur between an inlier/a gross 
outher(IO) and two gross outhers (00), hence, their corresponding averages 
of similarity values are smaller. 



3. Guided Sampling with Hypothesis Filtering 

This section describes our guided sampling scheme which involves a simul- 
taneous hypothesis filtering scheme. We also provide an efficient incremental 
update for computing the sampling weights. Furthermore, we provide an 
improvement on our guided sampling scheme based on the result from our 
hypothesis filtering scheme. 

3. 1 . Guided Sampling 

We use the similarity matrix K (Eq.|6]) to sample data in a guided fashion. 
Let Q = {su}u=i be the indices of data in a minimal subset of size where 
Su are indexed by the order in which they are sampled. The first element Si 
in Q is randomly selected from X. To sample the next element ^2, we use 
K{si^ :) as the weight to guide the sampling, i.e., the similarity values of all 
input data with respect to Si. We set K{si^Si) to to avoid sampling the 
same data again. Fig. [4] shows the example of such sampling weights. 
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Figure 4: Example of sampling weights. Left: two inliers from different structure. Right: 
their corresponding sampling weights for selecting the next datum. The data is arranged 
based on the structure membership (denoted by vertical dashed lines) for representation 
only. 

Suppose data 5i, • • • have been selected, then the next datum Su+i is 
chosen conditionaUy on the selected data. Its sampling weight is defined as 

K'{s^,:) ■ K'{S2,:) • ••• ■ K'{s^,:), (7) 

where • is the element-wise multiplication and K'[su-> :) is just K[su-> :) with 
K{su^Su) = 0. Eq. [t] means that in order to have higher probabilities of 
being sampled, a datum need to be similar (measured by Eq. [5]) to all the 
data that have been selected into the minimal subset. 

3.2. Incremental Top-k Lists Comparison 

Our sampling method computes an update to the similarity matrix K 
(Eq.|6]) once a block (of size b) of new hypotheses are generated. This involves 
comparing top-A: lists of ranked residuals. The computation of top-A; lists can 
be done efficiently via merge sort. However, comparing top-k lists between 
all data, i.e., constructing can be computationally expensive. Here we 
provide efficient incremental updates for K that can substantially accelerate 
the computation. 

As proved in [13j, the SF distance (Eq. [3]) can be equivalently computed 

as 

FW(r„r,-) = 2{k - \Z\)£ + \r^{m) - Tj{m)\ - r^{m) - J] r,M, 

(8) 



where Z = D^. n D^^ S = Dr.\D^ and T = Dr \D^.. In fact, S is simply 
the elements in D^. but not in Z, i.e., S = D^.\Z ^ similarly for T. Hence, we 
have 



n{m) = ^rn-Y^ n{m) = ^k{k + 1) - r,(m), (9) 

similarly for J2meT^ji^)- -^^ setting i = k + 1 and using Eq. |9| we can 
rewrite Eq. |8] to be in terms of Z only, 

F^'+'\Ti, T,) = {k + l)ik - 2 |Z|)+J](|r,(m) - r,(m)| + r,(m) + r,(m)) . 



(10) 



Let A and B be two N x N symmetric matrices and set the elements at 
the i-th row and the j^'-th column of A and B to 

A{iJ) = \Z\ and B{i,j) = (|r,(m) - r,(m)| + r,(m) + r,(m)) . (11) 

mez 



From Equations [5| [6| and [T0| the similarity matrix K can be constructed by 

= 2^) __L_B. (12) 



where 1^ Slu N x N identity matrix. Observe from Eq. [TTjthat the matrices 
A and B can be efficiently updated by keeping track of the elements that 
move into or out of Z. This information is readily available from the merge 
sort. Once A and B are updated, K can be updated via Eq. [T2| 



3.3. Simultaneous Hypothesis Filtering 

During sampling, we want to simultaneously filter hypotheses such that 
only a small but very promising subset remains. We make use the data 
preferences and hypothesis preferences to identify the good hypotheses. 

Similar to the definition of top-k hypotheses in Eq. |3| we define the top-h 
data of a hypotheses by sorting the residual information. For each hypothesis 

we sort its absolute residual rj = [r[^\r2 \ • • • , r^^] as measured to data 
and its sorted residual vector fj = [r^'^l)^ • • • ,'^^{1)] such that r^^^^^ < • • • < 

^1 ^1 
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r^''}j) . The top-h data of hypothesis 9j is defined as the first h elements in the 



permutation [n-^ 



,7rij^], i.e., 



(74 



(i) 



TT 



(i)i 



(13) 



The top-h data aj gives the /i data preferred by the hypothesis aj to be its 
inhers, i.e., the higher Xi is ranked, the more hkely Xi is the inher to it. The 
value of h is conservatively set to 0.1 in all experiments. The assumption is 
that at least 10% of data are inliers. 

For each hypothesis we construct the feature vector using the data 
and hypothesis preferences. 



Jj Jj 5 Jj 



E 



\E\ 



(14) 



where E = ^ i' and G Q^j} with = {xi G X | m G r^}, 

S = {(i, i' and G CTj} and K is the similarity matrix computed by 

Eq. [T2| The set contains all data that include the hypothesis j in their 
top-/c lists. If the hypothesis j is "good" , then Vt^ should contain many inliers 
from a structure. Hence, /j^^ the average of similarity values between all 
data in Vl^ should be high. Moreover, the top-/i data of hypotheses should 
contains data which are similar to each other (from the same structure), 
the average of similarity values between its top-/i data should be high, i.e., 
high /-^^ Therefore, we want to find a set of hypotheses which have high 



value in both and /)^^ To this end, we apply k-means on the feature 



p(2) 



vectors (Eq. 14) to separate "good" and "bad" hypotheses. As illustrated in 



Fig. [5](a), the cluster whose center has larger norm (circles) contains good 
hypotheses. We incrementally maintain a set of "good" hypotheses as the 
guided sampling proceeds. 

5.^. Selection of the First Datum in Minimal Subset 



The guided sampling scheme in Sec. |3.1| can be further improved by the 
result from the the hypothesis filtering scheme. 

In Sec. |3.1[ the first element in the minimal subset is selected randomly. 
The subsequent sampling the data of the minimal subset becomes unprof- 
itable if the first element is a gross outlier. Here, we can make use the 
result from hypothesis filtering to increase the probability of obtaining an 
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(a) (b) 

Figure 5: Hypothesis filteirng: (Left) Feature space where x axis is /^^ and y axis is fm"^ 
(Best view in color), (b) An example of "good" (denoted "g") and "bad" (denoted "b") 
hypotheses. 



inlier. Note that if a hypothesis is "good", its minimal subset should mainly 
contain inliers. Hence, we simply select the first element from the minimal 
subsets of the "good" hypotheses. 



4. Multi-Structure Fitting 

By leveraging the simultaneous hypothesis filtering, a set of "good" hy- 
potheses is immediately available once the sampling is done. This set of 
"good" hypotheses allows us to easily cluster the hypotheses using a simple 
agglomerative clustering method. 

We use the agglomerative clustering (See [14j for a detailed description) 
to cluster the hypotheses. For each hypothesis we aggregate the similarity 
of its top-h data as measured to data and represent each hypothesis 9j by 
a 1 X feature vector 

a, = 5]K(^,:). (15) 

The distance between two hypotheses 9j and 9j' is given by 

d{ej,ej>) = \\aj-aj4^, (16) 
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where ||-||2 denotes the L2 norm. The d{9j^ Oy) is smaUer if two hypotheses 6j 
and 9ji is similar, i.e, explaining the same structure. Using this distance mea- 
sure, the clustering is then performed through the standard agglomerative 
clustering mechanism. 

Each cluster of hypotheses contains the hypotheses overlapping on the 
same structure. For each cluster of hypotheses, we can simply select the 
hypothesis with the minimum of sum of squared residuals over its top-/i 
data. 

5. Experiments 

We test the proposed method on homography and fundamental matrix 
estimation using real data. To evaluate the efficiency of the proposed guided 
sampling scheme, we compare our method against 6 sampling techniques: 
Uniform random sampling in RANSAC (Random) [2j, proximity sampling 
(Proximity) PE], LO-RANSAC(LRANSAC) [15j, Guided-MLESAC (GM- 
LESAC) [4] and PROSAC [5]. Our proposed method is denoted by ITKSF 
(Sec. |3.1[ ) and its extension is denoted by ITKSF-S (Sec. |3.4D . 

In all experiments, we fix 6 = 100 and = [0.1 x t] throughout, h being 



the block size (cf Sec. 3.2) and t being the number of hypotheses generated 
so far. All experiments are run on a machine with 2.53GHz Intel Core 2 Duo 
processor and 4GB RAM. 

5.1. Data Set: AdelaideRMF 

We created a data set for robust model fitting, called AdelaideRMfQ It 
contains a collection of image pairs for homography and fundamental matrix 
estimation on single and multi-structure data. For each image pair, we use 
SIFT [16j to obtain the keypoint correspondences and manually labelled each 
keypoint correspondence. 

5.2. Homography Estimation 

This experiment involves estimating multiple planar homographies. The 
data used for this experiment is shown in Fig. [6| We use 4 correspondences 
to estimate a homography using Direct Linear Transformation [Ij. Each 
method is given 50 random runs, each for 5 CPU seconds. 



"'^ AdelaideRMF is publicly available from http : //cs . adelaide . edu. au/~hwong/doku.php?id=ciata 
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Table [T] shows the performance of guided samphng methods and also the 
hypothesis filtering result from our proposed method. 
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(c) Symon 




(d) NEEM 



91 69 58 X 78 

(e) Johnson 



Figure 6: Data for homography estimation. Red crosses indicate the gross outliers and 
other colored markers indicate the structure membership. (Best viewed in color) 
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Table 1: Performance of guided sampling methods in homography estimation. We record 
the number of hypotheses generated (M),the number of all-inlier minimal subset found 
on each structure (Structures) and the percentage of all-inlier minimal subsets (IS) found 
within the time budget. We also record the CPU time to hit at least one all-inlier minimal 
subset on each structure (HIT) which is penalized by the time budget if a method fails. 
For ITKSF and ITKSF-S, the hypothesis filtering result is shown in bracket. The results 
represent the average over 50 runs with the best result boldfaced. 
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5.3. Fundamental Matrix Estimation 

We also evaluate the proposed method on fundamental matrix estimation. 
The data used for this experiment is shown in Fig. [7| We use the 8-point 
algorithm [Ij to estimate a fundamental matri:x[^ Each method is given 50 
random runs and each for 10 CPU seconds. 

Table [2] shows the performance of guided sampling methods and also the 
hypothesis filtering result from our proposed method. 




□ 97 X 205 □ 63 O 102 X 77 

(a) Cube (b) Bread 




(e) ToysA (f) ToysB 



Figure 7: Data for fundamental matrix estimation. Red crosses indicate the gross outliers 
and other colored markers indicate the structure membership. (Best viewed in color) 



^http : //www . robots . ox . ac . uk/'^vgg/hzbook/code/ 
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Table 2: Performance of guided sampling methods in fundamental matrix estimation. The 
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6. Conclusions 



We propose a novel guided sampling scheme based on the distances be- 
tween top-k lists that are derived from residual sorting information. In con- 
trast to many existing sampling enhancement techniques, our method does 
not rely on any domain-specific knowledge, and is capable of handling multi- 
ple structures. Moreover, while performing sampling, our method simultane- 
ously filters the hypotheses such that only a small but very promising subset 
remains. This permits the use of simple agglomerative clustering on the sur- 
viving hypotheses for accurate model selection. Experiments on synthetic 
and real data show the superior performance of our approach over previous 
methods. 
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